1. Field of the Invention
The invention is a form of oculometer and as such may be used to measure the eye gaze direction and fixation duration, as well as the dual eye binocular convergence point. The invention has potential applications to the medical, scientific, engineering, manufacturing, military, and entertainment domains.
As examples, the invention may be used as a tool for the medical diagnosis of ocular functions, as an aid to the paraplegic handicapped, for the measurement of ocular functions and workload in human factors studies, as a measure of subject training, as a tool for fatigue monitoring, as part of an electronic safety net to detect performance degradation due to pilot incapacitation in piloted and teleoperated vehicles, as a component of an electronic intelligent pilot-vehicle interface used for adaptive aiding in piloted and teleoperated vehicles, for task scan analysis including measuring situation-awareness, for human operator control of machines and interaction with computer games, and for advertisement and usability analysis.
The invention is designed to be used with head-mounted video displays such as those that have been developed for virtual reality, stereographic displays, monocular or binocular vision helmet mounted displays, and night vision goggles. These displays are used in piloted helicopters, vehicles, and control stations for teleoperated robotics. The invention can be used as an eyetracker to control computerized machines from an electronic video display by the ocular gaze point of regard and fixation duration.
2. Discussion of Related Art
The present invention relates to U.S. Pat. No. 5,583,795, with the further improvements of: (1) a retinal scanning display as a light source, (2) a semiconductor, active-pixel image sensor array with integrated circuits for activating a charge coupled device (CCD) array, or in a further embodiment, a random access memory cache, to acquire an image of the human eye, and (3) expansion of the image processing to include inner structures of the eye and the retinal return.
The existing technology for oculometers is based upon the optical measurement of reflected infrared light from the human eye, and is more than forty years old in concept. In its present form, an oculometer contains a single infrared light source which is directed at the eye, and the reflected light is imaged onto a charge-coupled device (CCD) sensor array. The image of the eye is then electronically processed to determine the corneal reflection point and the pupil centroid. These parameters are used to determine the angular location of the eye relative to the camera within a fair degree of accuracy. The technology is either head-mounted or mounted in a panel in front of the user.
U.S. Pat. No. 5,583,795, referred to above, using a video display and an optical transistor sensor array, was an improvement in the existing technology. The video display is treated as a sequential array of light sources. The sensor array is integrated to a comparator array and an address encoder and latch, with both circuits clocked by the raster scan pulse of the display driver. This design is used to construct a data table pairing the sensors activated by specular reflections from the cornea to the sources as they are sequenced over the display field refresh cycle. Using ophthalmometric techniques, a three dimensional model of the external surface of the cornea is computed from the pairing table. The optical axis and the corresponding viewing direction are then computed from the model. In a further embodiment, the pupil center is located for computing the optical axis. This is done with ray tracing using the positions of the source and sensor paired by the reflection point located at the pupil centroid. Here, the centroid is determined from an image of the eye collected over the display field refresh cycle by a charge coupled device (CCD) array operating in parallel with the sensor array.
In this continuation of that patent, a retinal scanning display replaces the video display as a sequential source of light. In a further development, an improvement to the design of the sensor array is described for activating a CCD array, or in a further embodiment, a random access memory cache, to acquire an image of the eye. In a still further development, the optical-transistor sensor array and supporting circuits is embodied as an active-pixel image sensor array with integrated circuits, made from a complementary metal oxide semiconductor (CMOS). Finally, the image processing is expanded to include the inner structures of the human eye such as the pupil and iris, and the retinal return from the fundus.
Advantages Over Prior Art:
The light sources for the invention are produced by a retinal scanning display used as an imaging system in which the directed light sweeps the retina of the human eye for display effects. In this imaging system a modulated, coherent light sequentially illuminates adjacent point-wise portions of the human retina. The invention makes use of the shift in illumination of the human eye that occurs as the directed light scans in an orderly sequence.
The advantage of the retinal scanning display is that the eye is illuminated in exact, discrete steps. This generates a succession of precise reflection points from the cornea of the eye for processing by the sensor array of the invention. This is in comparison to the phosphorous video displays used in related U.S. Pat. No. 5,583,795 where successive elements of the display remain illuminated for a short time following activation resulting in relatively blurred reflection points.
Furthermore, the illumination from the retinal scanning display is brighter and more intense than that produced by the phosphorous video displays. This results in a more definite image of the pupil and an image return from the inner structures of the eye including the retina.
The invention uses an active-pixel image sensor array on a complementary, metal-oxide semiconductor (CMOS) substrate with integrated circuits in a parallel point array architecture. The design is readily manufactured as a very large scale integrated (VLSI) circuit array chip made from CMOS field-effect transistors (FET). The large scale of the design provides a resolution that is fine enough for accurate mapping of the cornea, and an image of the eye suitable for image processing. Furthermore, the CMOS VLSI array can perform at megahertz rates due to the circuit architecture. This is necessary for processing images at the raster-scan illumination rates of the retinal scanning display.
The point array architecture is a unique circuit design which ensures a highly accurate three dimensional mapping of the cornea. The point array circuit is a photo-diode, transistor integrated with a specular-threshold comparator and a bilateral switch. The switch performs as an element of a "winner-take-all" array for an encoder and latch circuit. The circuit is clocked by the raster-scan pulse of the display driver and determines the sensor element with maximum intensity above the specular threshold. A digital processor accesses the latched encoder to construct a corneal mapping table which pairs the directed light to the sensors activated by specular reflections from the cornea. This occurs as the directed light is sequenced over the display field refresh cycle.
The point array architecture employs novel circuits to acquire an image of the eye in a CCD array from diffused reflection without washout. This is necessary since the accumulation of specular reflections over a display refresh cycle would washout the image. At each raster-scan pulse, the outputs of all sensor array elements with intensities less than that for specular reflection are passed pixel-wise to the CCD. The output of each such element, as determined by a point-array specular-threshold comparator, drives a light-emitting-diode matched to a CCD array element through a bilateral switch. This is an improvement to a prior invention which provided no threshold limit.
In another embodiment, the sensor array acquires an image of the eye in a random-access-memory (RAM) cache. This embodiment, integrated with multiple memory caches, supports faster image processing. The outputs of the sensor-pixel elements drive point-array analog-to-digital converters. These output in turn to digital circuits for reading, adding, and writing to the RAM cache. Here, one cache can be downloaded while another is being loaded during a display field refresh cycle. The processing rate of a CCD array is limited by the bucket-brigade technique of downloading array values.
The invention employs a unique CMOS design which rapidly process the image acquired in the CCD array or RAM cache, at each display field refresh pulse. The image processor is embodied as a stack of very-large-scale integrated (VLSI) circuit arrays, which controlled by a central processing unit (CPU) is operated en masse along matched array elements. The stack design supports the application of advanced image processing techniques to isolate and enhance portions of the image, and to abstract the image coordinates of key features. These include the location and principal axes of the pupil centroid, and the image coordinates of the cluster points of the retinal capillary network following isolation from the retinal return. In addition, the image coordinates of key features of the inner structure of the human eye are computed such as the sphincteral pattern of the iris. The determination of these features provides the basis for a real-time three dimensional modeling of the human eye at the display field refresh rate.
The invention uses a novel method to compute the locations of features within the human eye from the corneal mapping table and the image coordinates accumulated over the display refresh cycle. The features are used to determine a three dimensional model of the human eye. In this method, an accurate model of the cornea is first computed from the listing of sensors to directed source lights paired by the digital processor. This model is used to locate the corneal center and the major and minor axes. With the corneal model, the method next computes the intraocular locations of features that are part of the inner structure of the human eye. This is done by first finding corneal reflection points which listed in the table are located at or near an image feature, interpolating a corneal surface refraction point, and then tracing an ophthalmic ray from that point back to the feature in the eye. This novel ophthalmomatric ray tracing method makes use of the corneal mapping to produce the optical locations within the eye of the key features isolated by the image processor.
The invention uses a best fit to a three dimensional model of the eye to compute the optical origin and the optical and median axes from the values for the corneal and internal features. These include the corneal optical center, the corneal surface center and axes, the pupil optical center, the pupil orientation, the capillary network of the retinal fundus, and the sphincteral pattern of the iris. Finally, the invention computes the viewing origin and axis for the eye from the optical origin and axis, and the median axes. The invention then locates the viewing origin and axis in the coordinate system of the retinal scanning display. The invention measures the visual gaze direction, point of regard, and the fixational duration from the location of the display in the workplace.
The invention is easily extended to the simultaneous tracking of both eyes allowing the measurement of the optical convergence point in the users three dimensional workspace either real or virtual.
The invention is auto-calibrating since the user can quickly and accurately go through a calibration procedure that correlates visual fixation position with line of sight.
The accuracy of the invention is independent of shifts of the helmet holding the display on the user's head; the shifts are caused by changes in facial-expression and head movement dynamics. This is due to the ability to compute an exact eye model from the locations of the light source and the sensors which are fixed by the helmet-construction. Measurements of the helmet location and orientation are used to relate the visual axes to the user's three dimensional workspace.